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SUMMARY 


Effectively  simulating  Synthetic  Aperture  Radar  (SAR)  requires  more  detailed  Digital  Feature 
Analysis  Data  (DFAD)  than  currently  provided  by  Defense  Mapping  Agency  (DMA)  standard 
Digital  Landmass  System  products.  Although  several  DFAD  prototypes  have  been  demonstrated 
to  support  SAR  simulation,  these  prototypes  cannot  be  produced  in  sufficient  quantities  to  meet 
simulation  requirements.  DMA  has  proposed  an  interim  high-resolution  DFAD  called  Level  3c, 
which  can  be  produced  in  quantity  using  standard  DMA  production  methods.  The  preliminary 
specifications  for  Level  3c  appear  to  meet  the  criteria  for  data  base  density  recommended  by 
previous  research.  At  the  request  of  the  Aeronautical  Systems  Division,  Deputy  for  Training 
Systems,  simulations  created  from  Level  3c  DFAD  were  compared  to  simulations  created  from 
two  other  prototype  DFADs  that  earlier  research  indicated  were  sufficient  for  simulating  current 
B-1B  SAR.  B-1B  Offensive  Systems  Officers  (OSOs)  performed  a  navigation  update  task  using 
both  simulated  and  actual  SAR  images  that  had  been  prerecorded.  The  dependent  variables 
were  accuracy  of  crosshair  placement,  operator  confidence  in  placements,  and  ratings  of 
acceptability  for  use  in  Weapon  System  Trainers.  The  results  demonstrated  that  SAR  simulations 
generated  from  Level  3c  supported  OSO  task  performance  equally  as  well  as  did  the  other 
high-resolution  DFAD  products.  We  therefore  recommend  that  the  Air  Force  accept  Level  3c 
as  an  interim  high-resolution  DFAD  product  for  simulating  current  SAR  systems  such  as  those 
on  the  B-1B  and  F-15E. 
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SIMULATION  OF  SYNTHETIC  APERTURE  RADAR  III: 
EVALUATION  OF  PROTOTYPE  DIGITAL 
LEVEL  3C  FEATURE  ANALYSIS  DATA 


I.  INTRODUCTION 

Simulating  Synthetic  Aperture  Radar  (SAR)  imagery  requires  Digital  Feature  Analysis  Data 
(DFAD)  with  greater  feature  density  than  is  currently  contained  in  standard  simulator  products 
from  the  Defense  Mapping  Agency  (DMA).  To  help  determine  DFAD  requirements  for  simulating 
SAR,  DMA  developed  a  prototype  high-resolution  DFAD  specification,  Level  X,  which  incorporated 
all  ground  features  10m  or  larger  on  a  side.  The  feature  density  in  Level  X  was  much  greater 
than  for  standard  products  such  as  Level  1,  which  has  a  100m  capture  criterion,  or  Level  2, 
which  has  a  30m  capture  criterion. 

An  engineering  analysis  (TASC,  1985)  concluded  that  Level  X  would  support  SAR  simulation 
for  Weapon  System  Trainers  (WSTs).  Production  of  Level  X  in  quantity,  however,  is  impractical 
due  to  the  coverage  required  by  the  major  commands  who  use  the  products  and  due  to  the 
large  number  of  man-hours  required  to  produced  Level  X  DFAD. 

Because  of  DMA’s  inability  to  produce  Level  X  DFAD  in  sufficient  quantities,  Headquarters 
Air  Force  tasked  the  Aeronautical  Systems  Division’s  Deputy  for  Training  Systems  (ASD/YW)  to 
define  an  alternative  DFAD  product  that  would  meet  the  SAR  simulation  needs  of  the  current 
B-1B  and  F-15E  WSTs.  ASD/YW  requested  the  Air  Force  Human  Resources  Laboratory’s 
Operations  Training  Division  (AFHRL/OT)  and  the  Aeronautical  Systems  Division's  Support 
Engineering  Division  (ASD/EN)  to  assist  them  in  defining  DFAD  requirements  for  simulation. 

Three  previous  experiments  were  conducted  by  AFHRL/OT  and  ASD/EN  to  define  minimum 
DFAD  requirements  for  SAR  simulation  (see  Crane,  Bell,  Kalinyak,  Dooley,  &  Hubbard,  1989, 
for  a  detailed  description  of  these  experiments).  In  Experiment  1,  SAR  simulations  were  generated 
using  the  high-resolution  prototype  DFAD,  Level  X,  with  a  10m  capture  criterion,  and  a  current 
DMA  product,  Level  2  DFAD,  with  a  30m  capture  criterion.  In  addition,  simulations  were  also 
developed  using  two  experimental  DFADs  with  15m  (Level  Y)  and  20m  (Level  Z)  capture  criteria. 
These  simulations  were  judged  for  acceptability  by  SAR-experienced  Air  Force  officers,  who 
indicated  that  a  capture  criterion  of  between  15  and  20m  would  produce  SAR  simulations  with 
acceptable  fidelity. 

Experiment  2  defined  the  ground  features  that  are  critical  for  performing  radar  scope 
interpretation  (RSI)  with  SAR  imagery.  The  features  most  often  identified  as  critical  by 
SAR-experienced  officers  were  lines  of  communication  (e  g.,  roads,  railroads,  and  canals)  and 
large  natural  features  (e  g.,  treelines,  shorelines,  and  cultivated  fields).  Nearly  all  of  these  critical 
features  are  included  in  DMA  Level  2  DFAD.  Small  individual  features  such  as  structures,  which 
are  included  in  Level  X  as  well  as  the  Level  Y  and  Level  Z  experimental  DFADs,  were  rarely 
cited  as  task  critical. 

The  results  of  these  experiments  appear  to  contradict  each  other.  The  major  conclusion 
drawn  from  Experiment  1  was  that  a  data  base  density  greater  than  current  DMA  Level  2  DFAD 
was  required  to  support  SAR  simulation.  The  results  of  Experiment  2,  however,  indicated  that 
RSI  is  based  primarily  on  the  large  features  and  lines  of  communication  contained  in  Level  2 
DFAD,  rather  than  the  small  individual  features  unique  to  Levels  X,  Y,  and  Z.  Based  upon  the 
comments  of  the  subject-matter  experts  in  Experiments  1  and  2,  it  appears  that  the  small 
individual  features  contained  in  Levels  X,  Y,  and  Z  function  primarily  to  provide  the  ground 
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detail  needed  to  approximate  the  familiar  level  of  image  clutter  within  the  simulated  SAR  image. 
If  this  is  indeed  the  case,  then  these  smaller  features  might  be  depicted  generically  without 
affecting  task  performance. 

This  hypothesis  was  tested  in  a  third  experiment  using  a  navigation  update  task  in  the  B-1B 
Engineering  Research  Simulator  (ERS)  at  Dyess  AFB,  Texas.  SAR  simulations  were  generated 
using  Level  X  (10m),  Level  Y  (15m),  Level  2  (30m),  and  Level  1  (100m)  DFAD.  The  Level  2 
and  Level  1  DFAD  were  enhanced  by  subdividing  high-dcnsity  areas  into  smaller  generic  features 
(see  Crane  et  a!.,  1989,  for  an  explanation  of  the  enhancement  procedures).  B-1B  Offensive 
Systems  Officers  (OSOs)  performed  a  navigation  update  task  using  these  simulations  and  actual 
SAR  imagery.  The  results  showed  no  significant  differences  in  either  the  accuracy  of  crosshair 
placement  or  the  OSOs’  confidence  in  their  placement  between  the  actual  SAR  imagery  and 
simulations  generated  from  Level  2  enhanced,  Level  Y,  and  Level  X. 

These  three  experiments  indicated  that:  (a)  lines  of  communication  and  large  natural  features 
are  the  principal  features  used  in  the  interpretation  of  SAR  imagery,  (b)  small  features  are 
required  to  provide  the  correct  SAR-like  appearance  in  high-density  areas,  and  (c)  these  small 
features  need  not  be  represented  with  ground  truth  accuracy.  Based  on  these  results,  an 
enhanced  Level  2  DFAD  was  recommended  as  being  acceptable  for  simulation  of  current  SAR 
systems. 

Unfortunately,  DMA  is  unable  to  produce  an  enhanced  Level  2  as  a  standard  product.  DMA 
proposed,  however,  that  a  high-resolution  DFAD  could  be  produced  by  digitizing  cartographic 
products  such  as  1:250,000  Joint  Operations  Graphics  and  1:50,000  topological  maps.  Called 
Level  3c,  this  DFAD  product  would  have  a  feature  density  similar  to  that  of  the  enhanced  Level 
2  but  would  use  fewer  separate  feature  codes  to  define  the  unique  characteristics  of  the 
individual  DFAD  features. 

The  objective  of  this  experiment  was  to  evaluate  the  effectiveness  of  Level  3c  DFAD  for 
simulating  SAR.  The  procedure  used  in  this  evaluation  was  similar  to  the  previous  performance 
evaluation  using  B-1B  OSOs  The  results  indicate  that  Level  3c  is  adequate  for  simulating 
current  B-IB-type  SAR  imagery. 


II.  METHOD 


Subjects 


The  subjects  were  B-1B  OSOs:  24  from  the  28th  BMW,  Ellsworth  AFB,  South  Dakota;  10 
from  the  319th  BMW,  Grand  Forks  AFB,  North  Dakota.  The  average  flight  experience  for  these 
OSOs  was  239  hours  in  the  B-1B  and  2,087  hours  overall. 


Task 

The  experimental  task  was  similar  to  that  used  in  our  previous  performance  experiment. 
Each  SAR-experienced  OSO  first  studied  Fixpoint  Graphics  of  nine  separate  aimpoints  in  the 
Strategic  Training  Range  Complex  and  then  performed  a  simulated  radar  navigation  update  over 
each  aimpoint,  using  either  simulated  or  actual  SAR  imagery  of  the  aimpoint  and  its  surrounding 
area.  After  each  radar  update,  the  OSO  rated  his  confidence  in  the  accuracy  of  his  crosshair 
placement.  Following  the  completion  of  all  nine  navigation  updates,  the  OSOs  rated  photographs 
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of  each  image  they  had  seen,  for  its  acceptability  as  a  SAR  simulation  to  support  WST  training 
tasks. 

The  navigation  update  task  was  simulated  on  the  B-1B  Engineering  Research  Simulators 
(ERSs)  at  the  28th  BMW  and  the  319th  BMW.  Only  the  radar  screen  and  track  handle  of  the 
OSO  station  in  the  ERSs  were  used  in  this  experiment.  The  SAR  images  of  the  aimpoints 
were  generated  off-line,  stored  on  video  disk,  and  presented  in  the  appropriate  order  to  each 
subject.  Since  the  ERS  uses  video  disk  to  store  images,  actual  as  well  as  simulated  SAR 
images  could  be  presented  in  the  simulator. 


Aimpoints 

A  subject-matter  expert  from  Strategic  Air  Command  (SAC/XPH)  selected  the  nine  aimpoints 
from  among  the  15  aimpoints  used  in  the  previous  task  performance  experiment.  The  coordinates 
of  each  aimpoint  are  given  in  Appendix  A. 


Data  Base  and  Image  Generation 

For  each  of  the  nine  aimpoints,  Level  2  enhanced  (2e),  Level  3c,  and  Level  X  DFAD  were 
obtained  from  either  DMA  or  from  ASD/ENETV.  SAR  data  bases  were  then  created  for  each 
combination  of  aimpoint  and  DFAD  level.  Twenty-seven  SAR  data  bases  (nine  aimpoints  x  three 
levels  of  DFAD)  were  built  for  AFHRL’s  Advanced  Visual  Technology  System  (AVTS).  Simulated 
SAR  images  were  then  generated  from  each  data  base  for  each  of  the  nine  aimpoints,  using 
the  SAR  simulation  capability  of  AVTS.  The  characteristics  of  the  AVTS  SAR  simulation  process 
are  described  by  Ferguson,  Ellis,  French,  Ball,  Spencer,  Bell,  and  Crane  (1989).  Each  simulated 
fixpoint  was  a  .625-  x  ,625-nautical  mile  (nm)  patch,  with  the  aimpoint  near  the  center  of  the 
patch.  For  each  simulation,  the  radar  illumination  angle  was  set  to  match  the  radar  illumination 
angle  of  the  actual  SAR  image  recorded  for  each  fixpoint.  All  imagery  was  generated  North-up. 

Each  aimpoint  was  simulated  from  two  depression  angles,  with  range  and  altitude  appropriately 
adjusted.  In  order  to  match  the  recorded  SAR  imagery  for  the  high  depression  angle  simulation, 
the  simulated  range  was  7-9nm,  with  an  altitude  of  8,000-10,000  feet  above  ground  level  (AGL). 
For  the  low  depression  angle  condition,  the  range  was  8nm  and  altitude  was  850  feet  AGL. 


Research  Design 

The  factorial  combination  of  the  nine  aimpoints,  three  DFAD  levels,  and  two  depression 
angles  resulted  in  54  simulated  SAR  images.  In  addition,  actual  high  depression  angle  SAR 
images  were  available  for  each  of  the  nine  aimpoints.  These  simulated  and  actual  SAR  images 
were  combined  to  create  the  63  SAR  images  used  in  this  experiment: 

Nine  Aimpoints  x  {(three  DFAD  Levels  x  two  Depression  Angles)  +  one  High 

Depression  Angle  Actual  SAR}  Images  Per  Fixpoint. 

The  experimental  design  was  an  incomplete  block  factorial.  Each  OSO  saw  each  of  the 
nine  aimpoints  once  during  the  experiment.  Seven  of  the  nine  aimpoints  represented  the  seven 
unique  combinations  of  image  source  (DFAD  Level  2,  3c,  X,  or  actual  SAR)  and  depression 
angle  (high  or  low,  with  only  high  depression  angle  for  the  actual  SAR).  The  two  remaining 


aimpoints  represented  the  replicates  of  various  image  source  by  depression  angle  combinations. 
The  presentation  of  various  fixpoint  by  image  source  by  depression  angle  combinations  was 
balanced  across  subjects  and  the  presentation  order  of  the  aimpoints  was  randomized  for  each 
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Procedure 


Briefing.  Each  OSO  was  briefed  individually.  The  purpose  of  the  experiment  and  the  type 
of  data  to  be  collected  were  described.  The  OSO  was  then  given  10  minutes  to  study  the 
Fixpoint  Graphic  cards  of  the  nine  aimpoints  plus  Fixpoint  Graphic  cards  for  two  additional 
aimpoints  used  as  warmup  trials. 

Warmup  Trials.  All  OSOs  received  two  warmup  trials  using  the  same  images;  performance 
and  confidence  data  were  recorded  but  not  retained  for  analysis.  After  both  warmup  trials, 
there  was  a  pause  for  the  experimenter  to  answer  any  questions. 

Data  Collection.  The  OSO's  task  was  to  locate  the  aimpoint  on  the  radar  screen  within  60 
seconds.  Before  each  trial,  the  OSO  was  given  30  seconds  to  review  the  Fixpoint  Graphic. 
At  the  end  of  the  review  interval,  the  screen  alphanumerics  were  illuminated  and  a  tone  sounded, 
indicating  that  the  OSO  should  request  a  radar  map.  When  the  map  was  completely  drawn, 
the  crosshairs  appeared  in  the  center  of  the  screen  and  a  response  clock  started.  The  OSO 
moved  the  crosshairs  to  the  aimpoint  by  depressing  the  track  handle  trigger  and  pushing  a 
thumb  control.  When  satisfied  with  the  crosshair  placement,  the  OSO  depressed  a  button  on 
the  track  handle  to  blank  the  screen  and  end  the  trial.  The  coordinates  of  the  crosshairs  at 
the  time  the  OSO  depressed  the  button  (indicating  that  he  had  located  the  aimpoint)  were 
recorded.  If  an  aimpoint  was  not  selected  within  45  seconds,  a  warning  tone  sounded.  If  the 
aimpoint  was  not  selected  within  60  seconds,  the  screen  blanked  automatically,  erasing  the 
image;  and  the  trial  was  terminated.  After  the  target  was  designated,  the  OSO  rated  his 
confidence  in  the  accuracy  of  his  crosshair  placement,  on  a  7-point  scale  ranging  from 
'1 -Complete  Guess"  to  "7-Very  High  Confidence."  The  experimenter  entered  this  rating  at  a 
terminal  and  started  the  study  period  for  the  next  trial.  After  performance  data  were  collected 
for  all  nine  trials,  each  OSO  was  shown  photographs  of  the  images  he  had  seen  in  the  simulator, 
along  with  the  appropriate  Fixpoint  Graphic.  The  OSO  examined  each  image  and,  by  selecting 
a  number  from  "0-Not  Adequate"  through  "4-Fuliy  Adequate,"  rated  the  image  as  to  its  adequacy 
for  WST  training  tasks. 


III.  RESULTS  AND  DISCUSSION 


Placement  Accuracy 


Crosshair  placement  accuracy  was  defined  as  the  radial  miss  distance  between  the  subject’s 
placement  of  the  crosshairs  and  the  ideal  placement  for  each  image  as  defined  by  the  Strategic 
Air  Command  subject-matter  expert.  To  make  variances  more  uniform  across  conditions  and 
to  reduce  the  relative  importance  of  a  few  very  poor  crosshair  placements,  a  logarithmic 
transformation  was  applied  to  the  miss  distances.  This  transformation  can  be  expressed  as: 

X'  =  Log2  (X  +  1) 

V  =  Log2  (Y  +  1) 

d  =  (X’2  +  Y’2)1/2 
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where  X  is  the  untransformed  miss  distance  on  the  horizontal  axis,  Y  is  the  untransformed  miss  distance  on 
the  vertical  axis,  X’  and  Y'  are  the  transformed  miss  distances  orr  each  axis,  and  d  is  the  radial  miss  distance 
used  in  the  analysis.  Figure  1  shows  the  mean  of  these  transformed  distances  as  a  function  of  image  source 
and  depression  angle,  in  arbitrary  units  forming  a  ratio  scale. 


Mean  Log  Miss  Distance 
4.5  - 


_ Low  Depression  Hl§§  High  Depression 

Figure  1.  Mean  Log  Miss  Distances  for  Levels  X,  3c,  2e,  and  SAR  for  High 
and  Low  Depression  Angles.  Miss  distances  are  in  arbitrary  units 
forming  a  ratio  scale.  Error  bars  are  95%  confidence  limits. 

The  miss  distances  were  statistically  analyzed  using  an  incomplete  blocks  model  analysis 

of  variance.  No  significant  effects  were  found  for  depression  angle  or  the  depression  angle 
by  DFAD  interaction  (p  >  .05).  Comparisons  of  simulated  and  recorded  SAR,  however,  must 
be  limited  to  the  high  depression  angle  condition  since  low  depression  angle  SAR  imagery  was 
not  available.  Overall,  placement  accuracies  for  Levels  X  and  3c  were  not  significantly  different 
from  each  other  and  were  significantly  more  accurate  than  placements  for  Level  2e  [F(2,  192) 

=  4.52,  p  =  .012].  At  high  depression  angles,  accuracies  for  SAR,  Level  X,  and  Level  3c 

were  not  significantly  different  from  each  other  and  neither  were  placement  accuracies  for  Levels 
X,  3c,  and  2e  significantly  different  from  each  other;  placement  for  SAR  was  more  accurate 
than  for  Level  2e  [F(1,  192)  =  4.18,  p  =  .042], 


Confidence 


OSO  confidence  in  placement  accuracy  was  rated  on  a  scale  of  1  to  7.  The  ratings  were 
analyzed  using  an  incomplete  blocks  model  analysis  of  variance.  Mean  confidence  ratings  are 
shown  in  Figure  2.  OSOs  were  more  confident  in  their  placements  with  high  depression  angles 
than  with  low  depression  angles  [F(1,  192)  =  10.20,  p  <  .01].  The  depression  angle  by  DFAD 
interaction  was  not  significant,  (F  <  1).  At  high  depression  angles,  confidence  for  Level  X 
was  higher  than  for  SAR  [F(1,  192)  =  10.81,  p  <  .001]  while  there  were  no  significant  differences 
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between  the  various  DFAD  levels.  Overall,  confidence  for  Level  X  was  higher  than  for  Levels 
3c  and  2e  [F(2,  192)  =  7.32,  £  <  .001). 


Mean  Confidence  Rating 


X  2e  3c  SAR 


Data  Base  Level 
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Figure  2.  Mean  Confidence  Ratings  for  Levels  X,  3c,  2e,  and  SAR  for  High 

and  Low  Depression  Angies.  Error  bars  are  95%  confidence  limits. 

Acceptability  Ratings 

OSOs  rated  each  image  for  acceptability  in  WST  training  on  a  scale  from  0  to  4.  The 
ratings  were  analyzed  using  an  incomplete  blocks  analysis  of  variance.  Mean  ratings  are  shown 
in  Figure  3.  High  depression  angle  images  were  rated  as  more  acceptable  than  were  low 
depression  angle  images  [F(1,  198)  =  18.11,  £  <  01).  The  altitude  by  data  base  level  interaction 
was  not  significant  (£  >  .10).  At  high  depression  angles,  SAR  images  were  rated  as  significantly 
less  acceptable  than  Level  X  [F(1,  198)  =  4.29,  £  =  .039],  whereas  ratings  for  3c,  2e,  and 
SAR  and  for  X,  3c,  and  2e  were  not  significantly  different  from  each  other.  At  low  depression 
angles,  ratings  for  Levels  X,  3c,  and  2e  were  not  significantly  different  from  each  other.  Overall, 
ratings  for  Level  X  were  higher  than  for  Level  3c  or  Level  2e  [F(2.  198)  =  9.67,  £  <  .01); 
ratings  for  Levels  3c  and  2e  were  not  significantly  different  from  each  other. 


OSO  Comments 

Several  of  the  OSOs  wrote  comments  on  the  rating  forms  regarding  specific  images.  Because 
these  comments  were  not  collected  systematically,  they  cannot  be  coded  for  statistical  analysis. 
However,  review  of  the  comments  shows  that  most  of  them  concerned  vertical  incidence  effects 
and  the  simulation  of  shadows  and  were  not  affected  by  data  base  level.  Comments  specific 
to  Level  3c  were  criticisms  of  how  storage  tanks  were  depicted  (i.e.,  they  were  often  too  small 
in  comparison  to  the  Fixpoint  Graphic  image).  The  AVTS  data  base  modelers  at  AFHRL  noted 
that  only  one  size  tank  Feature  Ident'f'c  ition  Descriptor  was  available  in  the  Level  3c  specification; 
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thus,  the  same  size  was  applied  to  all  storage  tanks  in  the  evaluation.  Tanks  of  varying  sizes 
were  used  in  the  Level  X  and  Level  2e  images. 


Mean  Rating 
4  - 


_ _  Low  Depression  HHI  High  Depression 

Figure  3  Mean  Acceptability  Ratings  for  Levels  X,  3c,  2e,  and  SAR  at  High 
and  Low  Depression  Angles.  Error  bars  are  95%  confidence  limits. 


IV.  DISCUSSION 


Data  Base  Density 


Our  previous  research  on  data  base  requirements  for  simulating  SAR  (Crane  et  al.,  1989) 
showed  that  the  high-density  ground  truth  information  characteristic  of  Level  X  (10m)  or  Level 
Y  (15m)  is  not  required.  That  is,  the  density  of  Level  2e  is  sufficient  to  support  SAR  task 
performance.  Counts  of  the  point,  line,  and  areal1  features  for  the  Level  X,  2e,  and  3c  data 
bases  used  in  the  present  experiment  show  that  the  density  of  Level  3c  is  equivalent  to  that 
of  Level  2e,  and  both  Levels  3c  and  3e  have  fewer  features  than  does  Level  X.  These  feature 
counts  are  shown  in  Figure  4.  Of  the  nine  scenes  used  in  this  experiment,  four  contained 
high-density  areal  features,  and  patterns  of  small  features  were  added  to  these  high-density 
areas  in  Level  2.  The  remaining  five  scenes  did  not  contain  high-density  areas,  and  the  Level 
2  data  bases  were  not  enhanced.  For  the  scenes  which  required  enhancement,  the  Level  3c 
DFAD  had  only  75%  of  the  features  found  in  Level  2e,  with  the  major  difference  relative  to  line 
and  areal  features.  However,  for  the  scenes  which  did  not  require  enhancement,  3c  had  an 
average  of  55%  more  features  per  scene  than  did  Level  2,  including  more  than  seven  times 


1  An  areal  feature  is  distinguished  from  a  point  or  line  feature  in  that  it  is  depicted  as  a  two-dimensional  surface  delimited  by  a 
surrounding  border. 
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the  number  of  point  features.  Overall,  the  density  of  3c  equals  that  of  2e  but  compared  to 
unenhanced  Level  2,  the  increase  in  density  is  more  uniform  across  scenes  and  is  therefore 
less  likely  to  show  areas  of  high  density  surrounded  by  low  feature  density. 

Feature  Count 
700  - 

600  -  ; - 1 


500  - 


Figure  4.  Mean  Feature  Counts  for  Point,  Line,  and  Areal  Features  in 
the  Nine  Scenes  Used  in  the  Level  3c  Evaluation  Experiment. 


Performance  Data 


Placement  Accuracy.  The  results  of  our  previous  OSO  task  performance  experiment  (Crane 
et  al.,  1989,  experiment  3)  cannot  be  directly  compared  to  the  results  of  the  present  experiment 
because  of  differences  between  the  image  sets  used  in  the  two  studies.  In  the  previous 
experiment,  there  were  15  scenes  divided  into  three  groups:  urban,  small  group,  and  isolated. 
In  the  present  experiment,  there  were  only  nine  scenes  in  a  single  group.  To  make  direct 
comparisons,  the  data  from  the  previous  experiment  have  been  reanalyzed  by  selecting  only 
those  aimpoints  included  in  both  experiments.  In  Figure  5,  the  mean  log  miss  distances  from 
the  earlier  experiment  are  compared  to  the  high  depression  angle  results  from  the  present 
experiment.  In  the  earlier  experiment,  SAR  operator  placement  accuracies  for  Level  2e,  Level 
X,  and  recorded  SAR  images  were  not  significantly  different  from  each  other.  In  the  present 
experiment,  crosshair  placements  at  high  depression  angles  were  significantly  more  accurate 
for  SAR  than  for  Level  2e.  Examination  of  Figure  5  shows  that  this  change  resulted  not  from 
a  decrease  in  performance  for  Level  2e  but  from  a  significant  improvement  in  placement  accuracy 
for  actual  SAR  images.  The  crosshair  placements  for  SAR  obtained  in  the  present  experiment 
were  not  only  more  accurate  than  for  Level  2e  in  this  experiment  but  also  more  accurate  than 
the  placements  for  Level  2e,  Level  X,  and  actual  SAR  from  the  earlier  experiment.  Placement 
accuracies  for  Levels  2e  and  X  did  not  change  significantly  from  the  first  experiment  to  the 
second. 

Confidence.  A  similar  analysis  of  confidence  ratings  shows  that  the  ratings  for  Levels  2e 
and  X  did  not  change  significantly  between  the  two  studies  (see  Figure  6);  however,  confidence 
ratings  for  SAR  decreased. 
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Figure  5.  Comparison  of  Placement  Accuracy  Results  from  a  Previous  OSO  Task 
Performance  Experiment  to  the  Level  3c  Evaluation  Experiment. 
Error  bars  are  95%  confidence  limits. 


Mean  Confidence  Rating 


I _ :  1st  Study  IllS  2nd  Study 

Figure  6.  Comparison  of  Confidence  Ratings  from  a  Previous  OSO  Task  Performance 
Experiment  to  the  Level  3c  Evaluation  Experiment. 


Summary 


Results  of  this  experiment  closely  parallel  those  obtained  by  Crane  et  al.  (1989),  except  for 
the  SAR  data.  For  SAR,  in  the  present  experiment,  placement  accuracy  increased  while 
confidence  ratings  decreased.  It  should  also  be  noted  that  acceptability  ratings  for  SAR  in  this 
experiment  were  lower  than  for  any  of  the  simulations,  although  the  difference  was  significant 
only  when  compared  to  Level  X.  No  explanation  is  offered  for  these  findings.  Placement 
accuracy  and  confidence  for  Level  3c  are  not  significantly  different  from  the  results  obtained 
for  Level  2e,  Level  X,  and  SAR  from  either  experiment. 


V.  CONCLUSIONS 

Data  Base  Density 

The  research  on  data  base  requirements  reported  by  Crane  et  al.  (1989)  concluded  that 
DFAD  with  sufficient  density  to  support  SAR  simulation  would  capture  as  many  features  as 
Level  2e.  The  density  of  features  in  Level  3c  is  less  than  that  in  Level  X,  greater  than  that 
in  Level  2,  and  similar  to  Level  2e.  Level  3c  also  has  a  more  uniform  distribution  of  features 
than  does  Level  2e.  This  increase  in  feature  density  was  achieved,  in  part,  by  compressing 
the  number  of  Feature  Identification  Descriptors  in  the  3c  specification  and  thus  reducing  the 
range  of  discriminations  that  might  be  made  regarding  feature  type,  height,  and  reflectivity. 


OSO  Task  Performance 

The  data  collected  on  placement  accuracy,  confidence,  and  ratings  of  acceptability  show 
that  Level  3c  DFAD  can  support  SAR  simulation  for  the  B-1B  WST.  Compression  of  feature 
identifiers  did  not  adversely  affect  operator  task  performance,  except  for  the  problems  noted 
regarding  the  identification  of  storage  tanks. 


VI.  RECOMMENDATIONS 

We  recommend  accepting  the  proposed  Level  3c  as  a  high-resolution  DFAD  product  to 
support  simulation  of  current  Air  Force  SAR  systems  such  as  the  B-1B  and  F-15E.  We  further 
recommend  that  DMA  continue  to  consult  with  Air  Force  subject-matter  experts  to  eliminate 
specific  problems  with  the  specification  as  these  problems  are  identified. 
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APPENDIX  A:  AIMPOINT  COORDINATES 


Level  X  radar 

(ixpoint  number  WGS  coordinates 


3 

44-52.44N 

104-09. 72 W 

4 

44-16.54N 

104-57. 95W 

5 

45-23.08N 

106-17.23W 

6 

44-29.32N 

1 08-03.32W 

7 

44-49.06N 

108-30. 34 W 

19 

46-20.88N 

102-37.22W 

20 

47-18.33N 

102-1 1.98W 

26 

45-44. 18N 

107-34.47W 

36 

45-23.50N 

101-25.84W 
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